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Introduction 


College and career pathways represent a significant national educational reform movement supported by 
federal, state, and philanthropic funders. High school pathways combine career and technical education 
with rigorous academics, work-based learning, and student supports to provide equitable access to 
postsecondary opportunities. Strong research evidence supports this specific combination of interventions 
as a means to interrupt the opportunity gap and address the underlying causes of disparate high school 
outcomes. Measuring Educational Quality and Improvement in College and Career Pathways was an 
opportunity to reflect on what is known about student outcomes in pathways, determine what is yet to be 
learned, and begin to propose research to accomplish that work. 


This symposium, the first of a planned series of four symposia focused on pathways research, is a joint 
effort by multiple research-based organizations active in policy development: College and Career Academy 
Support Network (CCASN), UC Berkeley; WestEd; SRI International; the California State University 
(CSU) Teacher Education programs, and the CSU Collaborative for the Advancement of Linked Learning; 
Jobs for the Future; the Learning Policy Institute, ConnectEd; the Linked Learning Alliance, PACCCRAS 
(Promoting Authentic College, Career, and Civic Readiness Assessment Systems) Working Group; and 
MDRC. 


Structure and Goals for the Day 


Forty-two (42) participants attended the day-long symposium, representing research and advocacy 
organizations and institutions of higher learning (see Appendix 1). The day was structured with plenary 
presentations on four strands of the student experience in college and career pathways, then subgroups by 
strand to map the current knowledge base, identify gaps, and prioritize focal questions for further research. 
After a policy overview at lunch, researchers then broke into new subgroups to collaborate and plan 
research projects. The agenda for the day is included in the program that follows this section. 


Based on their expertise, attendees had been assigned to one of the four student experience strands that were 
selected to begin the day. These were: 


¢ Data use in pathway interventions and design 

¢ Student achievement in pathways 

¢ Addressing opportunity gaps through student supports and access 
* Student engagement in pathways 


Attendees completed pre-work prior to the symposium, submitting bibliographies of research in their 
assigned strand. This work was compiled by strand as a foundation for subgroup work mapping the 
knowledge base to identify knowledge gaps. This annotated bibliography is included as Appendix 2. 


AGENDA 


8:00 —8:45 Registration & breakfast 


8:45 — 9:00 Welcome: Introduction to symposium & goals for the day 


9:00 — 10:00 Four plenary speakers outline the state of the field 


10:00 — 10:30 Plenary discussion of the strands for the morning session: 
* Student engagement in pathways 
« Student achievement in pathways 
« Addressing opportunity gaps through student supports and access 
« Data use in pathway interventions and design 


10:30 — 10:40 Coffee break & transition to break-out rooms 


Strand subgroup break-out session: Map existing studies & identify priority 


10:40 — 11:40 gaps in the research 


11:45 — 12:30 Strand subgroups report out 


12:30 -—1:30 Lunch speaker: California Assembly member Dr. Shirley Weber 


eee Plenary activity: Attendees reflect on patterns, overlaps, and opportunities 


presented by strand groups, and form new focal groups 


310 — 3:20 Afternoon break & transition to break-out rooms 


3:20-5:30 Focal working groups: Draft and post research proposals 


5:30 — 7:30 Wine reception and dinner buffet: Gallery walk — feedback and 
networking 


LUNCH KEYNOTE SPEAKER 


ASSEMBLYMEMBER SHIRLEY WEBER was elected in November of 2012 to represent 
California’s 79th Assembly District, which includes the cities of Bonita, Chula Vista, La Mesa, Lemon 
Grove, National City and San Diego. 


Born to sharecroppers of Hope, Arkansas, Shirley Weber has lived in California since the age of 3. 
She attended UCLA, where she received her BA, MA and PhD by the age of 26. Prior to receiving 
her doctorate, she became a professor at San Diego State University (SDSU) at the age of 23. Dr. 
Weber also taught at California State University at Los Angeles and Los Angeles City College before 
coming to San Diego State University. 


Dr. Weber chairs the Assembly Budget Subcommittee on Public Safety. She also serves as a member of 
the Assembly Standing Committees on Education, Higher Education, Elections, Budget, and Banking 
and Finance. 


KEYNOTE SPEAKERS 


KATHY BOOTH is a Senior Research Associate at WestEd where she translates research findings so that 
practitioners can apply them to pressing concerns and facilitates opportunities for 
community college leaders to develop action plans based on evidence. Her recent 
projects include developing the LaunchBoard — a data system that links educational, 
labor market and employment outcomes data; facilitating the creation of assessment 
competencies for the California Community College Common Assessment Initiative; 
and documenting employment outcomes for skills-builders — students who engage in 
short-term course-taking to advance their careers. 


Previously, Booth served as the Executive Director of the Research and Planning Group for California 
Community Colleges (RP Group). There she played a lead role in activities such as developing the 
Career and Technical Education Outcomes Survey, expanding research on the use of high school data 
to inform placement into basic skills courses, and documenting effective approaches to student 
support. Booth received a BA in women’s studies from Wesleyan University, with a focus on 
integrating diversity into progressive politics. 


DR. JAMES KEMPLE is the Executive Director of the Research Alliance for New York 
City Schools and Research Professor at the Steinhardt School of Culture, Education, 
and Human Development at New York University. As Executive Director, Dr. 
Kemple provides guidance and oversight for all Research Alliance work. He also 
serves as the Principal Investigator on a range of Research Alliance studies including 
those examining the efficacy of on-track indicators for different grade levels; 
performance trends in NYC high schools; and the effects of school closure. 


~ Prior joining the Research Alliance, Dr. Kemple spent more than 18 years at 
MDRC. He served as the Director of MDRC’s K-12 Education Policy Area and specialized in the design 
and management of rigorous evaluations, including randomized controlled trials of educational and other 
social policy reforms. He was the Principal Investigator for MDRC’s Career Academies Evaluation, as well 
as a number of other key studies. Dr. Kemple holds an Ed.D. and Ed.M. from the Harvard University 
Graduate School of Education, with a concentration in Administration, Planning, and Social Policy for 
Community and Urban Education, as well as a B.A. in Mathematics from the College of the Holy Cross. 


DR. TAMEKA McGLAWN isthe Chiefof Strategy, Learning, and Collective Impact 
for the Leading for Equity Collective Network, an organized, complex and 
dynamic leadership collaborative that builds coherence and alignment across 
systems to optimize performance, strengthen capacity and produce impactful 
results. She has provided servant leadership to learning and leading 
communities for nearly 25 years, and is considered a seasoned collaborator and 
evidence-based strategist and advisor. Having served ina myriad of settings and 
professional roles, Dr. McGlawn offers a unique perspective on implementing 
systemic and complex initiatives, tactical strategies to achieving equity-based 
outcomes and effective institutional and organizational practices. Her actionable research interests 
where she has served and facilitated professional learning and development include strategic reform 
implementation, systemic accountability and institutional equity, culturally responsive pedagogy and 
leadership, and building network alliances through innovative collective impact initiatives. 


Dr. McGlawn is a proud graduate of both the CSU San Diego’s Community Based Block Multicultural & 
Social Justice Master’s Program, and the Rossier School of Education at the University of Southern 
California where she earned her doctorate in Educational Leadership. She also has credentials in 
Administrative Services and Pupil Personnel. An effective consensus builder and action-oriented 
research practitioner, Dr. McGlawn brings an ideal synthesis of research savviness, practical application 
and a deep commitment to increased structural opportunities for all. 


DR. REGIE STITES is a senior researcher in SRI International’s Center for 
Education Policy. He has two decades of experience in the design and 
management of large-scale educational research and evaluation in the areas of 
literacy education, integrated academic and career-technical education, college and 
career readiness, and workforce development. Major projects include the 
Evaluation of the California Community College Linked Learning Initiative, the 
Evaluation of the California Linked Learning District Initiative, the Evaluation of the 
Oakland Health Career Pathways Project, and the Equipped for the Future National 
Work Readiness Credential Assessment Development and Validation. 


Stites holds a doctorate in education from UCLA, a master’s degree in cultural anthropology from the 
University of Wisconsin, a master’s degree in English/TEFL (teaching English as a foreign language) 
from San Francisco State University, and a bachelor’s degree in anthropology from the University of 
Illinois. 


Highlights from Key Presentations and Discussions 


The role of the key participants was to provide an overview of their strand, reflecting on critical research 
issues related to measuring the student experience. The role of the lunchtime keynote speaker was to 
provide a policy context for determining high-leverage research to pursue. 


Kathy Booth spoke on the challenges of data use in various contexts. Regarding school systems, data are 
not often provided to teachers and administrators in time to make any changes for the students from whom 
data was collected. Often educators have little control over how information is presented to them, nor do 
they necessarily get the level of granularity that they desire to contextualize information and make 
actionable decisions. Booth also noted that there is little reliable information on meta-cognitive and socio- 
emotional outcomes, college knowledge, or even on key experiences like work-based learning. Once 
students transition out of a school system, it is even more difficult to track students although longitudinal 
data systems are being developed by many states. 


Using data requires an institutional willingness to engage bad news, to change existing practices, to scale 
what works even when it requires reallocating resources, and to persist in reform efforts when they do not 
generate immediate results. Data usage is thus a cultural issue. 


Booth described some of the general barriers to answering important research questions around pathways, 
e.g., inaccurate information on which students are participating in which pathway and inadequate access to 
information on outcomes after the student has moved from one K-12 institution to another, enrolled in 
college, or entered employment. The challenges of obtaining and matching data include the difficulty of 
obtaining social security numbers, misunderstanding of FERPA requirements, and the fragility of data 
sharing systems developed through personal relationships. 


Dr. James Kemple began by defining student achievement broadly, including socio-emotional indicators as 
well as traditional academic ones. He proposed organizing and prioritizing outcomes by their proximity to 
the pathway components, following a theory of action. 


In education research, there is systematic variation in outcomes depending on non-malleable characteristics 
such as race, gender, and income. Some interventions are designed to “un-correlate” outcomes with 
background characteristics, but there is little evidence of the degree to which interventions like pathways 
can reduce that systematic variation. The priority should be to examine added-value outcomes—the 
distinctive influence that pathways have on outcomes over and above what students would likely have 
achieved in other high school alternatives that were available to them. This is important because pathways 
typically require additional resources, involve multiple actors and systems outside of the school system, and 
can be complicated and burdensome to implement well. These additional resources and efforts should be 
justified by the added value to outcomes in ways that alternative options are not likely to achieve. Given 
prior research indicating mixed impacts on educational attainment and achievement outcomes, and 
somewhat more consistent effects on labor market outcomes (when they are available), Dr. Kemple 
suggested placing a greater emphasis on labor market outcomes in an emerging theory of action. 


Dr. Tameka McGlawn addressed opportunity gaps and outcome gaps. She asked her colleagues to consider 
the role of the research community in disrupting patterns of inequity; to explicitly identify and call out 
structural policies and practices that perpetuate opportunity and outcome gaps. 


Research shows that when implemented with fidelity, Linked Learning pathways, California Partnership 
Academies, and NAF career academies show promise in providing expanded and equitable access to 
rigorous college and career preparation. However, there is considerable variation in implementation, 
suggesting a need for establishing and disseminating core goals and best practices. 


Dr. McGlawn argued for the importance of research-practice partnerships, student/family voice, and 
distributed leadership in college and career pathway research, practice, and policy. She suggested that 
methodologies such as DBIR (design-based implementation research) and PAR (participatory action 
research) are important qualitative tools, as valuable as “gold standard” methods (i.e., experimental designs) 
for making college and career pathway research useful to both policymakers and practitioners. Dr. 
McGlawn exhorted attendees to consider how researchers can help translate findings into measurable 
practice and action, create structures, and design policies that will eliminate opportunity and outcome gaps. 


Dr. Regie Stites discussed three topics around student engagement. First, the field must be able to define 
and measure student engagement. The definition must be multi-dimensional and include behavioral, socio- 
emotional, cognitive, and metacognitive factors. The indicators of engagement must be able to serve several 
purposes: formative feedback, continuous improvement, effectiveness and outcomes analysis, and 
accountability. 


Second, Dr. Stites suggested that the student choice of career theme, the integrated curriculum, and work- 
based learning experiences are pathways components that are associated with positive impacts on student 
engagement. Regarding work-based learning, research could help outline the barriers to student access to 
high quality experiences and could determine the optimum dosage, or range and number of work-based 
learning experiences, needed in order to engage all students. 


Finally, other features of pathways, such as the availability of industry-recognized credentials, should be 
explored for their value in increasing student engagement. Dr. Stites concluded by noting that pathways 
could serve as an effective model for engaging out-of-school youth, but for them, the standard 3- or 4-year 
pathway is not feasible. He challenged researchers to look at how specific components of pathways could 
be delivered to re-engage these youth. 


Dr. Shirley Weber, Assemblymember from California’s 79th Assembly District in the San Diego area, was 
the lunch keynote speaker. A former university professor and school board member, she serves on the 
education and higher education committees in the legislature and is known for her advocacy to close the 
achievement gap and provide an excellent education for all children. 


Dr. Weber began her discussion of the policy context of college and career pathways research with the 
intentionally provocative statement that research results actually matter very little in the decisions that 
policymakers make. She provided examples of education policies with very strong research behind them, 
such as the benefit of a later start time for high school students, yet legislators in Sacramento did not 
support such legislation--for political reasons. 


Dr. Weber’s comments invited the attendees to remember why pathways were developed in the first place, 
and for whom. She noted that sometimes an intervention shows promise and people want to scale it up, but 
a) it might not work for everyone, b) it was not designed for everyone, and c) scaling up might ignore the 
original population it was intended for. She cited a recent effort to get cybersecurity pathways funded, but 
no budget for outreach or inclusion efforts was included--in a field that is overwhelmingly male and white. 
It was not difficult for her to imagine which students would enter that pathway and be successful. Not every 
effort is a step forward in our efforts toward equitable outcomes in school. The measure of success of an 
educational intervention, she said, is how well it serves students who would not otherwise be successful. 


She concluded her remarks saying that policymakers need to be pushed to believe that they can make the 
kinds of changes that can improve schools. She exhorted the researchers in the room to go to their state 
capitols, push their agenda, and push it hard. Researchers need to use research to empower, engage, and 
build a movement that will settle for nothing less than equitable outcomes from schools. 


Plenary discussions after the key participants and keynote speaker invariably returned to issues of equity 
and how pathways could improve outcomes for traditionally underserved populations. Pathways were 
developed to provide equitable access to postsecondary and labor market opportunities, but outcome gaps 
remain. The speakers enumerated some of these gaps, as well as possible directions for evidence-based 
solutions. Equity thus became a central, overarching theme of the day as attendees asked whether the 
educational quality and improvement promised by pathways was being fulfilled equitably across population 
groups. 


Final Research Themes 


Once attendees had “mapped and gapped” the literature on college and career pathways and discussed the 
issues, they regrouped themselves according to potential work that could be done to address the gaps, and 
spent quality time developing collaboration and research proposals. They created new subgroups based on 
key research topics that emerged: 


¢ Engaging stakeholders through data 

* High quality college and career pathways 
¢ Measurement 

* Policy 

¢ Equity and access 


Working together in these subgroups, participants developed an overarching question related to their topic 
as well as individual research questions. A proposal template form was provided (see Appendix 3) in order 
to have a consistent final session. The form consists of a Proposal Development template for the 
overarching research question, and an Individual Proposal template for each research question generated 
that was developed into a proposal. The work of each subgroup is found in Appendix 4. 


The day resulted in a set of research topics and collaborative proposals that could lead to research to fill 
knowledge gaps. They represent the first step in what the Planning Committee hopes will be a continued 
collaboration among the participating organizations. Attendees left the symposium tired but appreciative of 
the deep conversations that were crystallized in the proposals. Each subgroup named a convener to schedule 
the next step, usually a conference call to further the ideas and discuss funding. 


New Organizational/Institutional Collaborations Developed 


Collaboration across organizations and institutions does occur, but not usually across as many as were 
represented in each subgroup formed at the symposium. In addition, at the time of the symposium, no 
funding for any of these projects was available. However, each attendee saw the value of the work, and one 
outcome was a commitment to pursue funding and continue collaborating. Prior to the break-out of the focal 
research subgroups, the following paragraphs were projected in the plenary room for discussion: 


Every organization here has dedicated tremendous resources to this work because we are committed to 
transforming education systems to be more equitable. But all of us are also to some extent competing for 
resources and we are also facing difficult times. As these working groups will be developing layered sets of 
research proposals around the strategic focal areas we’ve identified, we want to take a minute to address 
the question of who owns those proposals: 


By collaborating in developing these ideas, we intend to promote research in these high priority, high 
leverage areas, but we are also making a collaborative commitment to each other. Four organizations and 


five individuals may envision a project that interconnects with three different possible studies. One of those 
may be funded first for one organization. Another may be attacked with a doctoral study. We hope for a 
commitment from each of us here to work on opening up opportunities for each other, so that we can build 
on the work done today. We are asking for a commitment to join forces wherever possible to make what we 
plan here become a reality. We hope this symposium provides a foundation for working groups that will 
continue to meet to discuss logic models, share protocols, collaborate in the field, build upon each other’s 
data sets, and expand upon each other’s findings. 


The organizers of this symposium series would like this same sense of opening up new collaborative spaces 
to be present among funders too, including government as well as philanthropic sources, such that they 
entertain proposals from several parties. Indeed, funding sources should also consider collaborating on their 
giving such that a suite of related studies addressing a complex issue could be funded. 


As a result of attending the symposium, representatives from two participating organizations are currently 
engaged in conversation with a foundation official about a proposed new method for assessing the 21st 
Century skills that students display in project-based learning, internships, and other kinds of practice-based 
learning -- a hallmark of college and career pathways. These three new partners are developing a white 
paper and seeking additional funding for a pilot using new measures. 


All symposium participants and their organizations were encouraged to continue the commitments begun 
here to see fruition of their research ideas for the betterment of student outcomes in college and career 
pathways. Some of the topics developed in the first symposium will carry over into the second symposium, 
on equity issues in college and career pathway teaching and learning, and others will be picked up in the 
third symposium on capacity building. Collaborative research communities of practice that promote 
strategic research on college and career pathways were initiated at this first symposium, and are expected to 
continue to evolve over the course of the four symposia. 


Symposium Participant Feedback 


An online survey gathered feedback on the symposium. The response rate was 38% and reflected similar 
findings to those voiced in debrief conversations. In general, respondents indicated that the process used in 
the symposium was effective. (76% rated the process a 4 or 5, with 5 indicating “very efficient.” 23% rated 
the process a 3, with 0 respondents rating the process as | or 2.) Respondents were also satisfied with the 
outcomes of the day. (70% rated the outcomes 4 or 5, with 5 indicating “very satisfied.” 30% rated the 
outcomes as a 3, with 0 rating outcomes as a | or 2.) 


Process Feedback 
Overall, participants appreciated the organization of the day and the opportunity to discuss issues and 
collaborate with a group of well-informed scholars. 
¢  “T think that the day was well balanced between opportunities to learn, to collaborate, and to 
present individual ideas and perspectives.” 
¢ “Very well organized and productive day.” 
¢ “The process was efficient and focused.” 


Outcomes Feedback 
Participants noted that the most important outcomes of the day were the “opportunities to connect with 
colleagues and those working in the field,” discuss potential research topics, future collaborations, and 
research proposal development. Several participants also indicated that they benefited from connecting with 
prominent individual researchers and scholars in the field. 


¢ “Several of us came up with a potentially great cross-organization project idea...” 
¢ “The most valuable outcome for me was to develop a proposal.” 
* “Connecting with the individuals in the group was extremely beneficial. Many connections. Great 
participant list and perfect size group.” 
¢ “Overall, I thought this was a very valuable symposium on an important topic.” 
Suggestions for Future Symposia 
Most suggestions related to the pace and length of the symposium. There was a tension between trying to 
cover a substantial amount of material and opportunities for deep and thoughtful discussion. There was also 
a desire for more substantial framing of the broad purpose of the symposia and connecting the various 
activities that comprised the process of the day. 
¢ “The pace of the agenda constrained opportunities for sufficient discussion of the topics.’ 
¢ “My only suggestion for the next one is to simplify some of the group work tasks so that they can be 
completed in the available time.” 
° “Tt was very, very long.” 
¢ “Future symposia should be either a two-day event or a slower pace with a less intense agenda.’ 


’ 


, 


There were suggestions for improving the efficiency of the process in order to provide time for valuable 
discussions during the symposium: 
¢ “Proposal development process was overly ambitious given time - spend more time on priority 
topic identification instead and follow up after meeting.” 
¢ “Get more of the review of the literature done in advance, to get more quickly to the identification 
of high priority gaps. Also, build off of/integrate some of the proposals initiated in the first 
symposium.” 


Additionally, participants wanted a clearer sense of the expected commitment for future work beyond the 
symposium: 


¢ “Be clear at the outset what the expectations for work after the symposium will be. Provide an 
overview of what the whole process will entail. Provide more of a grounding in the motivation of 
the symposium and what it is supposed to accomplish (and how) over time.” 

¢ “I would have benefited from an explanation of the broader process that this first symposium was 
intended to start. There were various references throughout the day to what the focus of future 
symposia would be, but the broader process was not clear.” 


General Feedback 
One participant commented on the need for a greater social media presence such as a symposium hashtag, 
as a “vehicle for momentum and communicating how much this work means to so many partners.’ Many 
representatives of important educational institutions and organizations were in the room, and most of these 
organizations have very public and active social media handles to which these symposia need to link. 


Another participant was greatly interested in the role of teacher capacity to implement college and career 
readiness, indicating to the organizers the importance of the future planned symposia on teaching and 
learning and capacity building. “J think learning from teachers about issues related to their capacity to do 
everything on their plate (e.g., implement Common Core standards, prepare students for college and career 
readiness, mentor students, connect them to internships, provide work-based learning opportunities, work 
with higher education partners, take part in regional partnerships, prepare students for assessments, 
understand students' key cognitive strategies, etc.), specifically related to college and career readiness, 
would be fantastic. What kinds of professional learning opportunities have they found to be most valuable? 
What other supports do they need? Where are they stuck - or where do they need help the most?” 
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Many attendees appreciated the opportunity to set aside time to talk to colleagues and dig in to questions 
that are not often asked and engage in such a deep level of conversation. These include issues of equity and 
social justice which were raised by the key participants. 


Many attendees also expressed hope that these conversations would continue and despite everyone’s 
schedules, that the collaboration could continue. They noted that often there is excitement at such an event, 
but then it dissipates afterwards. There is lots of potential, one attendee said, “but it needs our attention.” 


Addressing the Feedback in Subsequent Symposia 
The chief take-away for subsequent symposia is the foundational role that equity issues play in pathways 
work. This has re-conceptualized subsequent symposia: Issues of equity need to be the lens through which 
all pathways topics are explored and proposals developed, throughout this symposium series. 


Symposium planners are currently exploring ways to increase attendees’ pre-work so that they arrive at the 
next symposium further along in the identification of research topics. This allows groups more time to 
develop research collaborations at the symposium. It is difficult to find such unrestricted time after the 
symposium, so more time is required during the day to create and refine these ideas. The conception of 
research collaboration was broadened beyond the development of specific proposals to also encompass 
sharing data, creating common metrics and definitions, and leveraging each other’s work. 
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Appendix 1: Participant List 


Keynote Speakers 


Name 


Title 


Affiliation 


Kathy Booth 


Senior Research Associate 


WestEd 


James Kemple 


Executive Director 


Research Alliance for NYC Schools, 
NYU 


Tameka McGlawn 


Chief of Strategy 


Learning and Collective Impact 
Leading for Equity Collective Network 


Regie Stites 


Senior Researcher 


Center for Education Policy, SRI 
International 


Shirley Weber 


Assemblymember 


CA Legislature 


Symposium Participants 


Monica Almond 


Senior Associate for Policy 
Development and Government 
Relations 


Alliance for Excellent Education 


Felicia Anderson 


Principal 


CSU CALL Fellow Long Beach 
Unified School District 


Kyra Caspary Senior Researcher Center for Education Policy, SRI 
International 

Anthony Dalton VP of Data Science Educational Results Partnership 

Neal Finkelstein Co-Director, Innovation Studies WestEd 

Edward Fletcher Associate Professor, College of University of South Florida 

Education 

Lynn Goldsmith Distinguished Scholar EDC 

Stephen Hamilton Professor Emeritus Cornell University 

Elisabeth Hensley Senior Research Associate RTI 

Jordan Horowitz Vice President Institute for Evidence-Based Change 
(IEBC) 

Catherine Imperatore | Research Manager ACTE 


Heather Lattimer 


Associate Professor, Learning and 
Teaching 


University of San Diego 


Will Marinell Director of Education Research One’ Foundation 
Nan Maxwell Senior Researcher Mathematica Policy Research 
Anna Ortiz Department Chair, Educational CSU - Long Beach 
Leadership 
Christine Padilla Senior Education Researcher Center for Education Policy, SRI 
International 
Bobbie Plough Associate Professor, Director, CSU - East Bay 


CRECE (Center for Research, 
Equity, and Collaborative 
Engagement) 


Russ Rumberger 


Professor Emeritus 


UC Santa Barbara 


Marisa Saunders 


Principal Associate 


Annenberg Institute for School Reform 


Katherine Shields 


Senior Research Associate 


EDC 


Patti Smith 


Director of Evaluation and 
Continuous Improvement 


NAF 


Cherie Solian 


Learning Coordinator, CSU CALL 
Fellow 


Lemoore Union ESD 
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Joel Vargas 


Vice President, School and 
Learning Designs 


Jobs for the Future 


Julian Vasquez Heilig 


Professor 


CSU - Sacramento 


Andrea Venezia 


Director/Professor 


EdInsights/CSU - Sacramento 


Mary Visher 


Senior Researcher 


MDRC 


Planning Committee Members 


Marisa Castellano 


Consultant 


Svetlana Darche 


Senior Research Associate 


WestEd 


Jay Fiene 


Dean, College of Education 


CSU - San Bernardino 


Anna Fontus 


Research, Data, and Policy Analyst 


Linked Learning Alliance 


Roneeta Guha 


Senior Researcher 


Learning Policy Institute 


Annie Johnston 


Coordinator of Public Programs 
and Principal Investigator 


CCASN, UC Berkeley 


Hueling Lee Consultant PACCCRAS (Promoting Authentic 
College, Career, and Civic Readiness 
Assessment Systems) Working Group 

Valerie Lundy- Associate Research Director Jobs for the Future 

Wagner 


Corinne Martinez 


Associate Professor, Teacher 
Education 


CSU - Long Beach 


Sara Pietrowski 


Director of Analytics 


Linked Learning Alliance 


David Stern 


Emeritus Professor of Education 


CCASN, UC Berkeley 


Laura Tobben 


Graduate Student 


CCASN, UC Berkeley 
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Appendix 2: Annotated Bibliography 


The following notes were submitted by attendees in advance of the symposium and compiled to help guide 
the discussions throughout the day. The entries are not intended to be complete summaries of the literature. 


STUDENT ACHIEVEMENT 


Carnevale, A. P., Rose, S. J., & Hanson, A. R. (2012). Certificates: Gateway to gainful employment and 
college degrees. Washington, DC: Georgetown University Center on Education and the Workforce. 
Retrieved from 

https://www.insidehighered.com/sites/default/server_files/files/06_ 01 12%20Certificates%20Full%20 
Report™%20FINAL.pdf. 


Key Relevant Findings: Potential of certificates and shorter credentials to facilitate employment and 
education 


Method: Survey mostly, but this area seems to be an area ripe for further study. The tremendous growth in 
micro-credentials that can be earned in college/career programs in high school is a potential area for study. 
Such programs are beginning to integrate such credentials into programs and research needs to assess their 
potential. 


Castellano, M., Richardson, G. B., Sundell, K. E., & Stone III, J. R. (2017). Preparing students for 
college and career in the United States: The effects of career-themed programs of study on high 
school performance. Vocations and Learning, 10(1), 47-70. 


Key Relevant Findings: Enrollment and participation in POS, CTE course sequences improved students’ 
probability of graduation by 11.3% and each additional CTE credit earned in POS increased their 
probability of graduation by 4%. There were non-significant effects for high school GPA. 


Method: Structural equations and an instrumental variable approach 


Additional Research Opportunities: | would really like to see outcomes studies disaggregated by program 
area because we would learn which programs contribute to the results more than others. Also, 
postsecondary data could be collected to examine post-POS effects. 


Castellano, M., Stone, R., Stringfield, S., Farley-Ripple, E.N., Overman, L.T., & Hussain, R. (2007). 
Career-Based comprehensive school reform: Serving disadvantaged youth in minority communities. 
Louisville, KY: National Research Center for Career and Technical Education. 


Key Relevant Findings: Researchers found neutral effects of career-based comprehensive reform efforts 
on the engagement, achievement, and transition of traditionally underserved students. Achievement: None 
of the three study schools achieved consistent gains over their comparison schools on measures of academic 
achievement. Engagement: The odds of dropping out declined as the proportion of the high school 
experience invested in CTE courses increased. Transition to Postsecondary: More students reported having 
a post-high school plan than their comparison peers at two of the three study schools. Many of the students 
at the study schools aligned their next step with their high school course of study. Regarding the 
achievement findings, researchers said: “as academic standards were being mandated at all three schools, 
CTE program areas had not yet had a chance to respond.” 


Method: This five-year longitudinal study included three feeder patterns of middle schools, high schools, 
and community colleges in communities with high percentages of at-risk students. High schools 
implemented career-based comprehensive school reform. High school engagement and achievement were 
measured using attendance, dropout, course-taking, and graduation data. High school transition was 
measured using responses to a senior survey, participation in Tech Prep and dual credit opportunities, and 
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achievement data for the graduates who attended their local community college. All measures were 
compared to the comparison school students not undergoing school reform. 


Castellano, M., Sundell, K. E., Overman, L. T., Richardson, G. B., & Stone, J. R. III. (2014, 
April). Rigorous tests of student outcomes in CTE programs of study: Final report. Louisville, KY: 
National Research Center for Career and Technical Education. 


Key Relevant Findings: Completing a POS was not associated with earning more college credits (e.g. dual 
enrollment) or an industry-recognized credential. Across all three districts, earning more CTE credits was 
associated with boosting the probability of graduation. Students in POS/career pathways outperformed their 
peers on the number of credits earned in STEM and AP classes, while also earning higher GPAs in their 
CTE classes. Both treatment and control group students planned full-time attendance at a four-year college 
at equal rates. The treatment group (POS/career pathways) however, indicated their studies would relate to 
their high school pathway. 


Method: The sample included 6,638 students from three urban districts in three different states. This study 
employed a multi-method, longitudinal, quasi-experimental design. Qualitative measures included 
adherence to the legislatively mandated components of POS. In the quantitative portion of the study, two 
different statistical approaches to the data were employed in each district. First, the authors estimated the 
effects of enrolling in POS and the number of career and technical education (CTE) credits earned on GPA 
and graduation using an instrumental variable approach. In addition, they also addressed specific policy 
questions about completing a POS compared to other high school trajectories through posthoc multiple 
regression analyses. 


Castellano, M., Sundell, K. E., & Richardson, G. B. (2017). Achievement outcomes among high school 
graduates in college and career readiness programs of study. Peabody Journal of Education, 92(2), 
254-274. doi:10.1080/0161956X.2017.1302220 


Key Relevant Findings: Outcomes (GPA, STEM credits) for high school students completing HS portion 
of CTE program of study (POS) were better than those of CTE concentrators who did not complete such a 
POS course sequence, and better than other students taking CTE courses outside a program of study. 


Method: Multiple regression analysis 


Additional Research Opportunities: (1) Additional analyses of the effects of POS participation that 
further reduce selection bias, and/or account for other relevant factors such as noncognitive attitudes & 
skills; (2) What are the mechanisms by which completion and concentration relate to differences in 
outcomes? (3) Authors suggest examining postsecondary and labor market outcomes associated with POS 
completion. 


Corrin, W., Sepanik, S., Rosen, R., & Shne, A. (2016). Addressing early waning indicators: Interim 
impact findings from the Investing in Innovation (i3) evaluation of DIPLOMAS NOW. New York, NY: 
MDRC. 


Key Relevant Findings: Program produced significant effects on students to stay above lower early 
warning threshold of 85 percent attendance, fewer than 3 suspensions or expulsions, and no fails in math or 
English. Program did not produce significant effects to stay above higher threshold of 90 percent 
attendance, no suspensions or expulsions, and no fails in any academic courses. Program did not produce 
significant effects on any single lower or higher threshold indicator. 


Method: Longitudinal random assignment 


Additional Research Opportunities: Need to wait for additional waves of data to determine impacts on 
graduation. 
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Dougherty, S. (2016) Career and technical education in high school: Does it improve student outcomes? 
Washington, D.C.: Thomas B. Fordham Institute. 


Key Relevant Findings: Dougherty found CTE course taking associated with higher 4-year high school 
graduation rates, employment rates, and earnings. He found greater effects for “concentrators,” students 
who took a sequence of 3 or more CTE courses. CTE Concentrators are 21 percentage points more likely to 
graduate high school than their identical non-concentrator peers. CTE Concentrators are 0.9 percentage 
points more likely to be employed and earn higher wages than similar non-concentrator peers the year 
following high school. CTE Concentrators are 1.3 percentage points more likely to be enrolled in a two- 
year college than similar non-concentrator peers the year following high school. 


Doughtery did not find evidence for significant tracking or overrepresentation of low-achieving students 
taking CTE courses. Low-income concentrators are 25 percentage points more likely to graduate than their 
low-income non-concentrating peers. Although he found graduation rate gains for males and female CTE 
concentrators, he only observed earnings gains for men. 


Method: Quasi-Experimental Design (QED; regression controlling for 8" grade achievement, attendance, 
disciplinary actions and demographics including free/reduced price meals eligibility) 


Additional Research Opportunities: The challenges related to obtaining labor market outcomes mean that 
the long-term impact of career academies on students’ career outcomes is understudied, aside from the 
Kemple study. Do early effects reported in this study persist? What accounts for the differential effects 
related to gender and SES? How does industry sector play into these relationships? What is the mechanism 
by which concentration affects outcomes? 


Kemple, J, J. (2008). Career academies: Long-term impacts on labor market outcomes, educational 
attainment, and transitions to adulthood. New York, NY: MDRC. 


Key Relevant Findings: Career academy students earned an average of 11% more annually than their non- 
academy counterparts and were more likely to live independently of their parents, 8 years after high school. 
No difference between the groups with respect to postsecondary educational attainment. In high school, test 
scores did not improve but academy students who entered high school with the highest risk of dropping out 
had better outcomes than similar non-academy students. 


Method: Longitudinal random assignment 


Additional Research Opportunities: Replication with larger numbers of sites, to answer: (1) Do some 
elements of the academy model have more impact than others, or do they work in concert? (2) How 
important is cohort scheduling? (3) What is the contrast between the academy and non-academy students’ 
experience now that there are more SLCs in high schools? 


Maguire, S., Freely, J., Clymer, C., Conway, M., & Schwartz, D. (2010). Tuning in to local labor 
markets: Findings from the sectoral employment impact study. Philadelphia, PA: Public/Private 
Ventures. Retrieved from http://ppv.issuelab.org/resources/5101/5101.pdf. 


Key Relevant Findings: Increased earnings for adults in programs aligned to local labor market needs 
Method: Experimental study of three relatively mature, sector-based programs 


Additional Research Opportunities: Research on sector-based strategies has all been on adult programs. 
Is it also a viable strategy for youth-based programs? 
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Maxwell, N. L. (2001). Step to college: Moving from the high school career academy through the 
four-year university. Evaluation Review, 25(6), 619-654. 


Key Relevant Findings: Career academy graduates from a large California district who entered a local 
university were more likely to complete their bachelor’s degrees, and less likely to need remediation, than 
other graduates from the same high schools. 


Method: Longitudinal comparison with statistical controls 


Additional Research Opportunities: (1) Do academy graduates earn higher wages while in college? If so, 
are they able to incur less debt? (2) What percentage of academy graduates choose majors related to the 
academy theme, and does that affect outcomes? 


Neumark, D., & Rothstein, D. (2007). Do school-to-work programs help the “forgotten half”? In 
Neumark D. (Ed.), Improving School-to-Work Transitions (pp. 87-133). New York, NY: Russell Sage 
Foundation. Retrieved from http://www.jstor.org/stable/10.7758/9781610444262.7 


Key Relevant Findings: Neumark and Rothstein examined initial postsecondary outcomes for six types of 
school-to-work programs: job shadowing, mentoring, co-op, school enterprise, tech prep, and 
internship/apprenticeship. Focusing on students in the bottom half of the distribution of predicted college 
attendance (“the forgotten half’), they found evidence for increased postsecondary education enrollment 
and earnings for men who participated in co-op, school enterprise, or internship/apprenticeship programs, 
and for women who participated in internship/apprenticeship programs. 


Method: QED (regression controlling for student demographics and prior achievement) 


Additional Research Opportunities: The challenges related to obtaining labor market outcomes mean that 
the long-term impact of career academies on students’ career outcomes is understudied, aside from the 
Kemple study. Do early effects reported in this study persist? 


Page, L. C. (2012). Principal stratification as a framework for investigating mediational processes in 
experimental settings. Journal of Research on Educational Effectiveness, 5, 215-244. 


Key Relevant Findings: Results provided suggestive evidence that school-sponsored exposure to the 
world-of-work serves as an important causal mechanism underlying the impact of the career academy on 
subsequent monthly earnings for males. 


Method: Principal stratification to find out the causal mechanism through which assignment to the 
treatment condition resulted in improved outcomes 


Additional Research Opportunities: This is an example of the after-the-fact work that can be done using 
data from quality studies (this study used MDRC/Kemple study data). 


Scrivener, S., Weiss, M. J., Ratledge, A., Rudd, T., Sommo, C., & Fresques, H. (2015). Doubling 
graduation rates: Three-year effects of CUNY's Accelerated Study in Associate Programs (ASAP) for 
developmental education students. New York, NY: MDRC. Retrieved from 
https://www.mdrc.org/sites/default/files/doubling graduation_rates_fr.pdf 


Key Relevant Findings: An accelerated developmental education model had large effects on students’ 
college persistence and degree completion. 


Method: Randomized-Control Trial 
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Additional Research Opportunities: The transition from high school to college for students needing 
additional preparation in basic skills is critical; studies like this one that assess the impact of interventions 
(such as ASAP, I-BEST, summer bridge programs, especially those that integrate with career pathways) are 
valuable and could be supplemented with more study of the role of specific program features. 


Stern, D., Saroyan, P., & Hester, C. H. (2012). Comparing students in each California Partnership 
Academy with non-academy students at the same high school, 2009-10. University of California, 
Berkeley. 


Key Relevant Findings: Within most schools, participation in a California Partnership Academy was not 
associated with differences in performance on the state standardized achievement test, but significant 
positive or negative relationships were found in some schools. 


Method: Multiple regression, including prior achievement scores, performed separately within each school 


Additional Research Opportunities: The variation in effects across academies is very interesting and 
could potentially be mined for various follow-up questions, as the authors note - e.g., What distinguishes 
the academies with positive significant results from those with no relationship to outcomes? Could the 
treatment effect be analyzed with additional mitigation of selection bias? 


Sun, J., & Spinney, S. (2017). Transforming the American high school experience: NAF's cohort 
graduation rates from 2011-2015. Fairfax, VA: ICF. 


Key Relevant Findings: Students who participated in NAF academies had graduation rates 3 percentage 
points higher than non-NAF students, 5 percentage points higher for at-risk students, and 6 percentage 
points higher for NAF students with full participation. This study replicates the SRI study’s findings that 
academies that show a high level of implementation fidelity have strong student outcomes. 


Method: Longitudinal quasi-experimental (propensity score matching) 


Additional Research Opportunities: There is a large database here that could be tapped more deeply but it 
lacks middle school pre-intervention baseline data. It’s hard to make achievement claims without that. How 
can we convince districts to provide more comprehensive hence useful data? More research needed with 
random assignment. 


Warner, M., Caspary, K., Arshan, N., Stites, R., Padilla, C., Patel, D., McCracken, M., Harless, E., 
Park, C., Fahimuddin, L., & Adelman, N. (2016). Taking stock of the California Linked Learning 
District Initiative. Seventh-year evaluation report. Menlo Park, CA: SRI International. 


Key Relevant Findings: Warner et al. found that students who entered certified Linked Learning pathways 
earned more credits and had higher graduation rates (5.3 percentage points) than similar peers in traditional 
high school programs, with larger effects for students who start high school with low levels of academic 
preparation. Based on a postsecondary survey conducted in a subsample of districts, they also found 
improved early employment outcomes (better job benefits) but found no difference in initial earnings or job 
autonomy. 


Students in certified and noncertified pathways had lower dropout rates, high graduation rates, and more 
credits than non-pathway students. Students in certified pathways earned slightly more college prep credits 
than non-pathway students but were not more likely to meet full requirements for UC/CSU or have higher 
GPAs than non-pathway students. Pathway and non-pathway students had similar rates of postsecondary 
enrollment and persistence. 


Method: Mixed method longitudinal evaluation; QED (regression controlling for student demographics and 
prior achievement) 
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Additional Research Opportunities: More research is needed on impacts of different LL programs—those 
in stand-alone schools, programs derived from former career academies, and new programs. I would really 
like to see outcomes studies disaggregated by program area because we would learn which programs 
contribute to the results more than others. Follow the longitudinal sample for several more years to 
determine eventual postsecondary educational attainment and career outcomes. The challenges related to 
obtaining labor market outcomes mean that the long-term impact of career academies on students’ career 
outcomes is understudied, aside from the Kemple study. Do early effects reported in this study persist? 


Xu, D., & Trimble, M. (2016). What about certificates? Evidence on the labor market returns to 
nondegree community college awards in two states. Educational Evaluation and Policy Analysis, 38(2), 
272-292. 


Key Relevant Findings: While finding positive impacts of non-degree credentials on earnings and 
employment, the study also documents variation in impacts both across fields, and across certificate 
programs within fields. 


Method: Individual fixed effects modeling using longitudinal wage record data 


Additional Research Opportunities: (1) State longitudinal datasets linking high school CTE participation 
to detailed degree and non-degree postsecondary credentials and wage records would allow for more 
nuanced analyses of CTE postsecondary & labor market outcomes (if available!); (2) What makes some 
certificate programs more effective than others? How do college program features play into the variation in 
results? (Kurlaender’s current work on CA community college CTE program policies related to student 
completion of credentials will be interesting on this topic: 
https://ies.ed.gov/funding/grantsearch/details.asp?ID=1692 


STUDENT ENGAGEMENT 


Advance CTE. (2017). The value and promise of career technical education: Results from a national 
survey of parents and students. Retrieved from 
https://cte.careertech.org/sites/default/files/files/resources/The_Value_Promise_Career_Technical_Ed 
ucation_2017.pdf 


Key Relevant Findings: Overall parents had a reasonably strong understanding and positive attitude 
towards CTE. 


Method: Focus groups and a national survey 


Additional Research Opportunities: Research is needed to better understand attitudes and awareness and 
how to overcome lingering stigma about CTE 


Allen, W. R., Kimura-Walsh, E., & Griffin, K. A. (2009). Towards a brighter tomorrow: College 
barriers, hopes and plans of Black, Latino/a, and Asian American students in California. Charlotte, NC: 
Information Age Publishing, Inc. 


Key Relevant findings: Several studies indicate that African American students are disproportionately 
underachieving as evidenced by numerous publications highlighting their educational upbringing as 
insufficiently resourced in culturally unresponsive K-12 schools. 


Although Linked Learning is embedded in several high schools, the achievement gap persists, African 
American students continue to be underrepresented in Advanced Placement courses, and overrepresented in 
special education. 


Method: Qualitative & document review of state data base 
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Additional Research Opportunities: Longitudinal research to identify the issues that African American 
students and their families deem relevant. 


Allensworth, E. M., & Easton, J. Q. (2007). What matters for staying on-track and graduating in 
Chicago public highs schools: A close look at course grades, failures, and attendance in the freshman 
year. Research report. Chicago, IL: Consortium on Chicago School Research. 


Key Relevant findings: This study identifies a variety of indicators—including students’ failures, 
absences, and overall grades—to learn what matters for a successful freshman year (and hence high school 
completion). 


The study provides evidence that academic behaviors (including student engagement) play a central role 
in determining students’ grades. Researchers looked closely at academic behaviors and their relationship to 
course grades and course failures for Chicago ninth-graders. Students’ prior test scores and background 
characteristics such as gender, race/ethnicity, economic variables, school mobility, and age at entry into 
high school together only explained12 percent of the variation in ninth-grade course failures. Students’ 
absences and self-reported study habits explained an additional 61 percent of the variation in 9" grade 
failures. Attendance and studying not only strongly predicted course failures but also were the strongest 
predictors for getting high grades—more so than test scores or student background characteristics. 


Findings from this study point to the importance of students’ course grades, grade point average (GPA), or 
class rank as vastly better predictors of high school and college performance and graduation, as well as a 
host of longer-term life outcomes, than their standardized test scores or the coursework students take in 
school. With this finding, the researchers conclude that students’ academic preparation for high school is 
far less important for simply passing courses than is their behavior in high school, particularly their course 
attendance. Course passing rates are primarily determined by attendance. Students attend class more often 
when they have strong relationships with their teachers, and when they see school and their coursework as 
relevant and important for their future. 


Method: This report analyzes correlations between student performance in freshman coursework, eventual 
graduation, and personal and school factors which may contribute to success of failure in freshman-year 
courses. 


Bragg, D. D. (2007). Teacher pipelines: Career pathways extending from high school to community 
college to university. Community College Review, 35(1), 10-29. 


Key Relevant Findings: One case using targeted support around Accuplacer support saw a 25% decline in 
remediation to the community college (CC). Two cases showed a large increase in grade-to-grade retention; 
though in one case that increase may not be attributed to the intervention. One case doubled the number of 

students matriculating to the CC. Authors note that these outcomes are mid-point measures as the programs 
had only been operating for 2-3 years, making the impact on baccalaureate degree attainment impossible to 
assess. 


Method: Multiple case study with 3 sites implementing teacher pathways 


Additional Research Opportunities: The impact of articulated HS to CC to Univ. pathways; dual 
enrollment, targeted support for college placement tests. 
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Cabrera, N., Miner, D., & Milem, J. (2013). Can a summer bridge program impact first-year 
persistence and performance? A case study of the new start summer program. Research in Higher 
Education, 54(5), 481-498. 


Key Relevant Findings: New Start Summer Program (NSSP) participation was a significant, positive 
predictor of both first-year retention and GPA after controlling for entering student characteristics, but it is 
likely that the most significant effects of NSSP participation are indirect. 


Method: The research team derived data for this project from two sources: The Office of Institutional 
Research Planning and Support (OIRPS) at the University of Arizona and a longitudinal survey developed 
by the research team. 


Additional Research Opportunities: Do findings extend to graduation? Are there other programs that can 
replicate these results that might take place in the senior year? 


Cooper, K. S. (2014). Eliciting engagement in the high school classroom: A mixed-methods 
examination of teaching practices. American Educational Research Journal, 51, 363-402. 


Key Relevant Findings: Identified connective teaching, academic rigor, and lively teaching as strong 
predictors of student engagement, with connective teaching being most predictive. Study introduces a 
typology for thinking systematically about teaching for engagement. 


Method: Case studies of classrooms, factor analyses of practices, and multilevel regression analyses 


Additional Research Opportunities: Some career courses are included in the analysis but not the case 
studies. Work could start from here, focus on pathways classes, and begin to link engagement to 
achievement. 


Dougherty, S. M. (2016). Career and technical education in high school: Does it improve student 
outcomes? Washington, DC: Thomas B. Fordham Institute. 


Key Relevant Findings: CTE concentration dramatically improves graduation rates but has modest effects 
on other outcomes 


Method: Analysis of Arkansas database 


Additional Research Opportunities: Is CTE concentration in anything a good thing? Or are there some 
fields where concentration is not so good? 


Engberg, M. F., & Wolniak, G. C. (2013). College student pathways to the STEM disciplines. 
Teachers College Record, 115(1), 1-27. 


Key Relevant Findings: Students who report greater preparedness in HS math were more likely to declare 
STEM majors, indicating that dual enrollment programs where students take college level math stands to 
increase the number of students declaring STEM majors. Authors note this may be especially the case for 
under-represented students “by linking them directly to apprenticeships and easing the burden of later 
college costs” (p. 23). 


Method: Analysis of 2002 ELS dataset with 2006 follow up. Cross-classified hierarchical general linear 
modeling 


Additional Research Opportunities: Long-term effects of dual enrollment and enhanced math preparation 
for all students 
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Fletcher, Jr., E. C., & Tyson, W. (2017). A longitudinal analysis of young adult pathways to STEMH 
occupations. Career and Technical Education Research, 42(1), 35-55. doi:10.5328/cter42.1.35. 


Key Relevant Findings: Science course-taking, math GPA, and science GPA did not predict entering 
STEMH occupations, but Calculus did. Women entered at lower rates than would be expected given HS 
achievement. SES did not predict technician or professional STEMH attainment when HS achievement was 
controlled. Participation in STEMH CTE in HS had stronger results for PSE enrollment, degree attainment, 
vocational certifications and obtaining STEMH professional careers than a traditional HS college prep 
curriculum. 


Method: 1997-2009 National Longitudinal Survey of Youth; HLM 


Additional Research Opportunities: Given the growth in pathways since 1997, this study should be 
replicated. Can we do this in the 2002-2017 time frame? 


Guha, R., Caspary, K., Stites, R., Padilla, C., Arshan, N., Park, C., Tse, V., Astudillo, S., Black, A., & 
Adelman, N. (2014). Taking stock of the California Linked Learning District Initiative. Fifth-year 
evaluation report. Menlo Park, CA: SRI International. 


Key Relevant Findings: SRI’s 5" year evaluation of the Linked Learning District Initiative focused on 
students: Who enrolls in pathways? How do students feel about their experiences? What are their 
perceptions of the skills they are gaining? And, what effect does participation in a Linked Learning pathway 
have on students’ high school outcomes? Researchers used two measures to assess student engagement: 
attendance and retention within the district. They found: 


(1) In general, average attendance rates for pathway and non-pathway students were high. No 
evidence that students who enrolled in certified pathways had better attendance than similar peers in 
traditional high school programs. 


(2) On average, students enrolled in certified pathways were 2.2 percentage points more likely to 
stay within their district from 9th to 10th grade, 4.6 percentage points more likely to stay through 11th 
grade, and 5.2 percentage points more likely to remain through 12th grade, compared with similar peers in 
traditional high school programs. 


Method: Multi-method research design that includes qualitative and quantitative data collection and 
analysis 


Harper, S. R. (2009) Institutional seriousness concerning Black male student engagement: Necessary 
conditions and collaborative partnerships. In: Harper, S. R. and Quaye, S. J. (eds.) Student 
Engagement in Higher Education: Theoretical Perspectives and Practical Approaches for Diverse 
Populations [pp. 137-155]. New York, NY: Routledge. 


Harper, S. R., & Davis III, C. H. (2012). They (don't) care about education: A counternarrative on 
Black male students' responses to inequitable schooling. The Journal of Educational 
Foundations, 26(1/2), 103. 


Key Relevant Findings: Too often the onus is put on the African American student to assimilate; the 
responsibility to bridge the divide must be on the institution to shift their pedagogical practices to match the 
needs of the disenfranchised. 


Students reported pathway practices that include a peer buddy system to support connectedness as an 
integral part of the freshman experience. 


Method: (1) This chapter reviews recent research and highlights from national survey data regarding Black 
male student engagement. (2) This article is based primarily on a systematic content analysis of essays 
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written by 304 Black male undergraduates attending colleges and universities across the United States. Each 
applied to participate in the University of Pennsylvania's Grad Prep Academy, an initiative for college 
juniors who aspire to earn Ph.D.s in the field of education. 


Additional Research Opportunities: More research could be done to explore other key factors that 
potentially support connectedness 


Harper, S. R., & Quaye, S. J. (2009). Beyond sameness, with engagement and outcomes for all: An 
introduction. In S. R. Harper & S. J. Quaye (Eds.), Engaging racial/ethnic minority students in 
predominately white classroom environments [pp. 1-12]. New York, NY: Routledge. 


Key Relevant Findings: These authors explain that the alienation, marginalization, and isolation that many 
African Americans students experience in school appears to be directly related to their lack of engagement. 
To counterbalance these harsh realities, the school community could intentionally focus on culturally or 
ethnically-based student organizations, cultural awareness, and student support services to potentially affect 
positive mediation. 


Method: (1) This paper examines student perceptions of a novel human physiology laboratory format and 
the effect of prior experience on these perceptions. The same undergraduate human physiology course, 
taught at second year level, was taken by students who had previously completed a semester of human 
physiology (‘continuing’ students) and by those taking it for the first time (‘new’ students). Continuing and 
new students were compared using a correlational analysis. 

Additional Research Opportunities: Research on multiple high school sites asking the same questions 
regarding African American student engagement and achievement as indicated by graduation rates and 
readiness for college and/or careers. The data could be compared across sites to tease out the nuances of 
quality implementation, systemic support, and institutional intentionality. Additionally, a qualitative study 
could be done to focus on teachers to inform best practices, educator apathy, retention, student advocacy, 
and professionalism. 


Kemple, J. J., & Snipes, J. C. (2000). Career academies: Impacts on students’ engagement and 
performance in high school. New York, NY: MDRC. Retrieved from 
https://eric.ed.gov/?id=ED441075 


Key Relevant Findings: This study explores the extent to which career academies can alter the high school 
environment in ways that better support students academically and developmentally, and how the context in 
which career academies are implemented influence their effects on student outcomes. Findings suggest that 
increasing both the level of interpersonal support that students experienced during high school and their 
participation in career awareness and work-based learning activities can substantially improve high school 
outcomes among students at high risk of dropping out. For this group, the academies reduced dropout rates, 
improved attendance, increased academic course-taking, and increased the likelihood of earning enough 
credits to graduate on time. Note that when averaged across the diverse groups of students and sites 
participating in the study, academies produced only modest improvements in students’ engagement and 
performance during high school. 


Method: Large-scale, multi-site, random assignment research design 


McCharen, B. (2008). The success of implementing programs of study in health careers through 
career clusters and pathways. Career and Technical Education Research, 33(3), 203-215. 


Key Relevant Findings: 54% (lower than state average) of HS program completers continued to 2 or 4- 
year IHEs in health-related programs. Of those who entered certificate programs, 78% completed them. 
Only 2% completed degrees at the IHEs. 


Method: Descriptive, longitudinal study using institutional data, follow-up surveys, and phone calls. 
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Additional Research Opportunities: The study used unemployment insurance data files for outcome data- 
how can these data be used elsewhere? 


Moore, C., Venezia, A., Lewis, J., and Levkovitz, B. (2015). Organizing for success: California’s 
Regional Education Partnerships. Sacramento, CA: EdInsights. 


Key Relevant Findings: This report describes the following strategies suggested by representatives of the 
studied partnerships: 


¢ Start with a coalition of the willing, 

* Focus on a few main goals, 

¢ Foster involvement of key leaders, 

° Leverage existing networks, 

¢ Create opportunities for cross-system communication and collaboration, 

¢ Embed activities in existing organizations, and 

¢ Use data to motivate action and inform activities. 
The report concludes with suggestions to help state policy and philanthropic communities better support 
regional partnerships. 


Method: The information used in this report came from 37 interviews with stakeholders from 19 regional 
consortia located throughout California. 


Additional Research Opportunities: More research could be done to test/expand upon the authors’ 
recommended strategies (viability, challenges, opportunities, effects on student outcomes), and to provide 
exemplars. 


National Research Council. (2004). Engaging schools: Fostering high school students’ motivation to 
learn. Washington, D.C.: National Academies Press. 


Key Relevant Findings: The National Research Council and Institute of Medicine (2004) summarized 
decades of research to identify school conditions that promote strong student engagement and positive 
academic mindsets. These included: presenting students with challenging but achievable tasks; 
communicating high expectations for student learning and providing supports that allow students to meet 
these expectations; making evaluation practices clear and fair and providing ample feedback; reinforcing 
and modeling a commitment to education and being explicit about the value of education to the quality of 
one’s life; providing students with opportunities to exercise autonomy and choice in their academic work; 
requiring students to use higher-order thinking to compete academic tasks; structuring tasks to emphasize 
active participation in learning activities rather than passively “receiving” information; emphasizing variety 
in how material is presented and in the tasks students are asked to do; requiring students to collaborate and 
interact with one another when learning new material; emphasizing the connection of schoolwork to 
students’ lives and interests and to life outside of school; and encouraging teachers to be fair, supportive, 
and dedicated to student learning while holding high expectations for student work. 


Method: Descriptions of various school conditions and analysis for patterns promoting student engagement 
and positive academic mindset 


Page, L. C. (2012). Understanding the impact of career academy attendance: An application of the 
principal stratification framework for causal effects accounting for partial compliance. Evaluation 
review, 36(2), 99-132. 


Key Relevant Findings: Author estimated an average causal effect of treatment assignment on subsequent 
monthly earnings of approximately $588 among males who remained enrolled in an academy throughout 
high school and more modest impacts among those who participated only partially. 
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Method: Re-analysis of Kemple 2008 career academy study using a principal stratification framework and 
Bayesian inference to investigate impacts. 


Additional Research Opportunities: More research is needed to unpack the drivers of positive impacts of 
academies. 


Phelps, L. A., & Chan, H. (2016). Optimizing technical education pathways: Does dual-credit course 
completion predict students’ college and labor market success. Journal of Career and Technical 
Education, 31(1), 61-85. 


Key Relevant Findings: Completing an average of 6 units of Transcripted Credit and Advanced Standing 
dual enrollment courses was related to college success and annual earnings at age 22. Enrollment in longer, 
diploma granting manufacturing and engineering programs resulted in better labor market outcomes than 
shorter term and health and other non-STEM programs. Better preparation in math (as measured by higher 
Accuplacer scores) was related to better labor market outcomes. Dual enrollment offered on the HS 
campuses, taught by HS CTE instructors predicted greater levels of college success and labor market 
outcomes. 


Method: HLM using data from K-12 data systems with unemployment insurance wage records in 
Wisconsin 


Additional Research Opportunities: Are there similar results for baccalaureate degree outcomes? The 
kind of dual enrollment course, location, and instructors are important in this study- how about for other 
contexts? 


Swanson, J. L. (2008). An analysis of the impact of high school dual enrollment course participation on 
post-secondary academic success, persistence and degree completion (Order No. 3323472). [Doctoral 
dissertation]. University of Iowa. Retrieved from 
https://search.proquest.com/docview/304633260?accountid=14496 


Key Relevant Findings: Purpose: To investigate the impact of high school students' participation in dual 
enrollment courses upon college persistence and degree attainment. * Dual enrollment participants were 
11% more likely to persist through second year of college and were more likely to enroll in Post-Secondary 
Education directly after HS than were non-participants. 


Method: Inferential statistics on a casual model developed for this study 


Additional Research Opportunities: Do dual enrollment students graduate in higher numbers? 


Theodos, B., Pergamit, M. R., Hanson, D., Edelstein, S., Daniels, R., & Srini, T. (2017). Pathways after 
high school: Evaluation of the Urban Alliance High School Internship Program. Washington, DC: 
Urban Institute. Retrieved from https://theurbanalliance.org/wp- 
content/uploads/2017/08/08292017.pdf 


Key Relevant Findings: Students in Urban Alliance internships performed better than a control group that 
did not experience internships: 


¢ Increased the likelihood of young men who’ ve gone through the program attending college by 23 
percentage points 

¢ Increased the likelihood of middle-tier students (2.0 — 3.0 GPA) enrolling in a 4-year college by 18 
percentage points 

¢ Increased students’ comfort with and retention of critical professional skills over time, especially 
young men. 


Method: Random assignment 
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Additional Research Opportunities: Additional rigorous studies of internships are important to see if 
these results are replicable. 


¢ Research is needed on other WBL models, such as youth apprenticeships 

¢ Research is needed to better understand the correct “dose”: how many hours, pay versus no pay, 
type of tasks, connection to classroom learning 

¢ Research is needed to compare traditional internships with alternate models such as school based 
enterprises and virtual internships 


Vargas, J., and Venezia, A. (2015). Co-design, co-delivery, and co-validation. Boston, MA: Jobs for the 
Future. Retrieved from www.jff.org/publications/co-design-co-delivery-and-co-validation. 


Key Relevant Findings: This report focuses on defining and “unpacking” what it means to co-design, co- 
deliver, and co-validate curricula and other components of pathways across systems. It also outlines barriers 
to those approaches. 


Method: This is a thought piece that draws from the authors’ own research (over the past 20+ years) and 
from a literature review. 


Additional Research Opportunities: More research could be done to test the authors’ suggested 
framework (viability, challenges, opportunities, effects on student outcomes), and to provide exemplars. 


Visher, M. G., Altuna, J. N., & Safran, S. (2013). Making it happen: How career academies can build 
college and career exploration programs. New York, NY: MDRC. 


Key Relevant Findings: Three-year study of the implementation of the ECCO program. ECCO was 
launched in 18 career academies in six school districts in three states: California, Florida, and Georgia. The 
purposes of the study are to document the experiences of these schools in adopting the program and to 
assess the extent to which, when given support and resources, programs like ECCO can be fully 
implemented. 


Method: Descriptive data were collected to assess the promise to improve student participation in career 
and college exploration activities and to improve their awareness of postsecondary options. 


Additional Research Opportunities: Focus on work-based learning activities in career academies, 
documenting student experiences and engagement in specific work-based learning activities. 


Woolley, M. E., Rose, R. A., Orthner, D. K., Akos, P. T., & Jones-Sanpei, H. (2013). Advancing 
academic achievement through career relevance in the middle grades: A longitudinal evaluation of 
CareerStart. American Educational Research Journal, 50(6), 1309-1335. 


Key Relevant Findings: The authors hypothesized that the CareerStart program would promote teachers’ 
use of career relevant instruction, which in turn would promote student behavioral, emotional, and cognitive 
engagement in school and learning while ultimately advancing student academic achievement. A significant 
treatment effect was found for math performance, but not for reading. 


Method: Cluster randomized trial (CRT). Seven of 14 middle schools were randomly selected to be either 
treatment schools (receiving intervention) or control schools. Analysis was conducted as a rigorous intent- 
to-treat model comparing student outcomes for students attending a treatment school with those who 
attended the control schools. 


Additional Research Opportunities: Focus on middle school programs, but the study hypothesis that 
career-relevant instruction across subject areas contributes to student engagement/academic achievement 
can be adapted for programs in high schools. The study suggests in its next steps that additional school 
staff, librarians, and counselors should also be fluent in career-relevant content. 
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Issues did arise based on the transitory nature of both students and teachers across schools. 


OPPORTUNITY GAPS 


Berger, A., Turk-Bicakci, L., Garet, M., Knudson, J., & Hoshen, G. (2014). Early college, continued 
success. Early college high school initiative impact study. San Mateo, CA: American Institutes for 
Research. 


Key Relevant Findings: 81% of early college students enrolled in college, as compared with 72% of 
comparison students. 25% of early college students earned a degree (mainly an Associate’s degree). No 
effect of early college on college enrollment rates for students from different backgrounds; however, effects 
on degree attainment were stronger for minority and low-income students, and those with higher prior 
achievement. 


Method: Longitudinal, lottery-based random assignment 


Additional Research Opportunities: It would be instructive to extend this study in the following ways: 1) 
examine effect of early college on labor market outcomes; 2) estimate the cost-savings (and increased 
earnings) of attending early college; 3) identify the components of early college high schools that raise 
student outcomes; and 4) explore which of these components appear to be driving impact. 


Carnevale, A., Garcia, T., Gulish, A. (2017). Career pathways: Five ways to connect college and 
careers. Center on Education and the Workforce. Washington, DC: Georgetown University. 
Retrieved from https://cew.georgetown.edu/wp-content/uploads/LEE-final.pdf 


Key Relevant Findings: This is a conceptual publication that makes an argument for the importance of 
aligning pathways with labor market demand, aligning curriculum with workforce requirements, offering 
high quality counseling, and conducting job placement and skills gap analyses. 


Method: Conceptual piece rather than research; includes case studies. References lots of labor market 
research. 


Additional Research Opportunities: Additional qualitative research is needed to identify how schools, 
districts, states, etc. have implemented college and career pathways that have had a positive impact on 
student outcomes. There is a broad gap in organizational and policy research about how to implement these 
programs. 


Castellano, M. E., Richardson, G. B., Sundell, K., & Stone, J. R. (2016). Preparing students for 
college and career in the United States: The effects of career-themed programs of study on high 
school performance. Vocations and Learning, 10(1), 47-70. doi: 10.1007/s12186-016-9162-7 


Key Relevant Findings: POS enrollment improved students’ probability of graduation by 11.3% and each 
additional CTE credit earned increased graduation probability by 4%. 


Method: Structural equation modeling and instrumental variable approach 


Additional Research Opportunities: Tracking students to examine whether they chose majors and 
pursued careers in occupations related to their POS. 


ea 


Constantine, J. M., Seftor, N. S., Martin, E. S., Silva, T., & Myers, D. (2006). Study of the effect of the 
Talent Search Program on secondary and postsecondary outcomes in Florida, Indiana, and Texas. Final 
report from Phase II of the National Evaluation. Washington, DC: U.S. Department of Education. 


Key Relevant Findings: The Talent Search Programs (which typically include career advising), visiting 
college campuses, and developing students’ knowledge of financial aid assistance, had positive effects on 
students staying in and completing high school 


Method: Analysis of administrative data compiled in 3 states; a quasi-experimental design to create 
matched comparison groups 


Additional Research Opportunities: More research to explore the role of access to college and career 
counselors who can provide information about postsecondary options in keeping students in school 


Hill, L. D. (2008). School strategies and the “college-linking” process: Reconsidering the effects of 
high schools on college enrollment, Sociology of Education, (81)1, 53-76. 


Key Relevant Findings: Found three (3) distinctive college-linking strategies: Traditional: School offers 
limited resources and limited organizational commitment to facilitating students’ access to resources. 
Clearinghouse: Fairly substantial school structure of resources but weak organizational commitment. 
Brokering: Exceptional structure of college resources plus norms promoting equitable distribution of 
resources to students and families. 


Method: Quantitative study; used two-level multinomial, hierarchical models to measure school strategies 
that facilitate college enrollment and students’ postsecondary outcomes 


Additional Research Opportunities: Examine school structures and types of resources that promote 
college enrollment, particularly among traditionally marginalized populations 


Holland, N. E. & Farmer-Hinton, R. E. (2009). Leave no schools behind: The importance of a college 
culture in urban public schools, High School Journal (92)3, 24-43. 


Key Relevant Findings: Students in small schools/small learning communities perceive greater access to 
and involvement in school’s college-going culture, e.g. more college-prep instruction and activities, in- 
depth conversations with staff, encouragement and support from staff, interactions with counselors, and 
hands-on, school-based college/career planning activities. 


Method: Quantitative study: used descriptive analyses, ANOVA, and correlations of school size and 
college support measures 


Additional Research Opportunities: Investigate how students’ race and socioeconomic status exclude 
them from college and career preparation due to school tracking structures, funding, and resource inequities, 
and private-public school disparities. 


Institute for Entrepreneurship in Education. (n.d.). Creating alternative pathways to postsecondary 
education: Da Vinci Schools Extension Program. San Diego, CA: School of Leadership and Education 
Sciences, University of San Diego. Retrieved from 
https://drive.google.com/file/d/OB6QNXPbGJG2HLUdaSUoyV UpBVWe/view 


Key Relevant Findings: Program evaluation study found that structured academic opportunities, intensive 
mentoring, and internships provided strengthened student focus and interest in postsecondary persistence. 


Method: Interviews with students in program and comparison group 
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Additional Research Opportunities: There are a range of “wrap around” style programs that are 
supporting students in the postsecondary transition. Although some have had individual evaluations, more 
needs to be done to look at these programs holistically. 


Johnson, J., & Rochkind, J. (2009). With their whole lives ahead of them: Myths and realities about why 
so many students fail to finish college. New York, NY: Public Agenda. Retrieved from 
https://www.publicagenda.org/files/theirwholelivesaheadofthem.pdf 


Key Relevant Findings: The need to work while attending college, financial challenges, and limited 
information are all factors that have a significant impact on students who leave college before graduating. 


Method: Public Agenda surveyed more than 600 young adults, ages 22 to 30, who had at least some higher 
education coursework. Those who started college but did not complete a degree were asked why they left, 
and their views, experiences, and responses were compared to those of students who had successfully 
completed a two- or four-year college program. 


Additional Research Opportunities: Student voice remains limited in discussions of strengthening college 
access and success. 


Kautz, T., Heckman, J. J., Diris, R., Ter Weel, B., & Borghans, L. (2014). Fostering and measuring 
skills: Improving cognitive and non-cognitive skills to promote lifetime success (No. w20749). National 
Bureau of Economic Research. 


Key Relevant Findings: This is a synthesis of literature related to how to foster and measure the types of 
non-cognitive skills that are necessary for success in the workplace. 


Method: Literature review/research synthesis 


Additional Research Opportunities: This article—along with many related pieces by James Heckman, 
Martin West, and others—make a compelling case for the important role that non-cognitive skills play in 
affecting students’ long-term interests (and in how to measure these skills). What we need now are studies 
that investigate the impact of various college and career pathways on the development of these skills. 


DATA 


Booth, K. (2015). Moving the needle: Data, success, and accountability for workforce programs. 
Oakland, CA: WestEd. 


Key Relevant Findings: Describes challenges experienced by California community colleges and the 
Chancellor’s Office in collecting and acting on information regarding completion and economic outcomes, 
including large numbers of metrics with diverse definitions, missing information on certificate programs 
and other key data points such as work-based learning, lack of data sharing, inability to interpret data, and 
information presented in a fashion that makes it difficult to use. 


Method: Examination of how career and technical education (CTE) success is captured and documented in 
the California community college system, to provide context for recommendations from the field. 


Data Quality Campaign. (2015). Preparing students for jobs: Ensuring student success in the 
workforce. Washington, DC: Author. 


Key Relevant Findings: To develop and support a strong workforce, policymakers, educators, school and 
business leaders, students, and the labor force can use data to inform improvements to the variety of routes 
through education and careers. 
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States can securely link limited K-12 data with postsecondary and workforce data, such as program 
completion or employment status, to evaluate which schools, programs, and pathways help students be 
successful in college and careers. 


States can also collect and report information on students’ readiness for college and careers to provide 
transparency and inform decisions about how to best prepare students for college and beyond. 


Method: a series of analyses that highlight state progress and key priorities to promote the effective use of 
data to improve student achievement. 


Hughes, K. L., & Karp, M. J. M. (2004). School-based career development: A synthesis of the literature. 
New York, NY: Community College Research Center, Teachers College, Columbia University. 


Key Relevant Findings: These studies, well-regarded for their strong methodology and useful findings, 
give evidence that, as a whole, career guidance interventions can positively influence students’ career 
development. Guidance activities directed at junior high school students had the largest effect sizes. 
Students do seem to benefit, both vocationally and academically, from participation in career courses. In 
particular, they seem to increase their knowledge of careers and their ability to make career-related 
decisions. On most career-related measures, students did see increased outcomes when compared with 
students not enrolled in a career course. However, there is little evidence that any gains—either academic or 
career-related—are maintained over time. In most cases, no follow-up research was conducted over time to 
see the lasting nature of any knowledge gain or attitude change, or the relationship of these with actions 
taken later. 


Method: Literature review 


Additional Research Opportunities: More research is needed on career development in lower grades, and 
on the lasting impacts of career development activities. 


Moore, C., Bracco, K. R., & Nodine, T. (2017). California’s maze of student information: education 
data systems leave critical questions unanswered. Sacramento, CA: Education Insights Center. 


Key Relevant Findings: California collects expansive sets of data about students in its public K-12 and 
higher education systems— data that, collectively, have great potential to meet the information needs of 
state policymakers, local educators, and other stakeholders. But the data are collected and maintained in 
systems that are not connected, were designed for different purposes, are subject to different regulations, 
and often use different data definitions. As a result of these disconnects, important information about 
student progress is often impossible to access, share, and use—whether at the state, regional, or local level. 
While there may be a few benefits to the current structure, they are outweighed by major disadvantages, 
including inefficient use of taxpayer dollars. 


Method: Qualitative research 


Savard, R., & Michaud, G. (2005). The impact of LMI on the career decision-making process: 
Literature review. Quebec, Canada: University of Sherbrooke. 


Key Relevant Findings: Most studies that discuss labour market information (LMI) as it relates to career 
decision-making report on the positive impact of computer-assisted programs, but these were carried out 
before the Internet became a part of our information environment. 


Eveland, Conyne and Blakeny (1998) used DISCOVER to measure the participants’ level of career 
decidedness. The results showed that using the DISCOVER program had a significant impact on increasing 
the participants’ level of career decidedness when the computer was combined with a counsellor (F = 
7.1000 p= <0.05). However, the results were less significant when the computer was used without the 
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guidance of a counsellor (F = 3.3548 p = <0.05). There was no significant increase in career decidedness in 
the control group (F = 0.2069 p= <0.05). 


Roney (1999) concluded that it is insufficient to simply transmit information for people to make realistic 
decisions. He also found that the stronger the students’ academic performance and social commitment, the 
better they are able to evaluate labour market information. 


Method: Literature review 


Additional Research Opportunities: More research is needed that looks at types and methods of 
delivering labor market data and its impact on career decision-making. 


Steven, H., Matthew, L., & Collen, P. (2016). Building better bridges to life after high school: 
Experimental evidence on contemporary career academies. Proceedings of research symposium, 
“What Shapes a Student’s Path Forward?”: The tenth annual conference on the National Center for the 
analysis of longitudinal data in educational research (CALDER). Washington D.C. 


Key Relevant Findings: Enrollment in the career academies under study increases the likelihood of high 
school graduation and college enrollment, each by about 8 percentage points, with the attainment gains 
concentrated among male students. Academy participation reduces 9th grade absences but has little 
influence on academic performance, AP course-taking, or AP exam success during high school. Analysis of 
candidate mechanisms suggests that roughly one fifth of the overall high school graduation effect can be 
attributed to improved student engagement in high school. 


Method: This paper examines the profile of students entering career academies in one school district and 
estimates causal effects of participation in one of the district’s well-regarded academies on a range of high 
school and college outcomes. 


Warner, M., Caspary, K., Arshan, N., Stites, R., Padilla, C., Patel, D., McCracken, M., Harless, E., 
Park, C., Fahimuddin, L., & Adelman, N. (2016). Taking stock of the California Linked Learning 
District Initiative. Seventh-year evaluation report. Menlo Park, CA: SRI International. 


Key Relevant Findings: The seventh annual evaluation report finds: compared with similar peers in 
traditional high school programs, students participating in certified Linked Learning pathways were more 
likely to graduate from high school, less likely to drop out, and earned, on average, more credits. In 
addition, certified pathways were doing just as well as traditional high school programs at helping students 
complete the a—g requirements even as they retained more students who might otherwise have dropped out 
and were unlikely to pursue the full college preparatory curriculum. Certified pathway students were as 
likely as similar peers in traditional high schools to enroll in college. Conditional on enrollment in any 
postsecondary institution, pathway students were also equally likely to enroll in a 4-year college and to 
persist in school to a second year. Although the finding for enrollment ina4 year college is not significant 
in the overall sample, it is significant and positive for African American students and students with low 
prior achievement. Finally, one year out of high school, pathway students were more likely than their non- 
pathway peers to obtain jobs with benefits such as vacation and sick leave. 


Method: Continued analysis of longitudinal data from SRI International’s multiyear evaluation of the 
California Linked Learning District Initiative, including postsecondary outcomes for students in Linked 
Learning pathways. 
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Western Interstate Commission for Higher Education. (2014). Building capacity for tracking human 
capital development and its mobility across state lines. Policy Insights. Retrieved from 
http://www.wiche.edu/info/publications/PI-150105-MLDE-Policy%20Insights.pdf 


Key Relevant Findings: Combining data across just the four multistate longitudinal data exchange 
(MDLE) states helps to reveal employment outcomes for 9.6 percent of the college graduates who would be 
invisible in Washington, to 22.4 percent who would otherwise not be found in Idaho. 


51 percent of the first-time college student cohort in the participating four states were from out of state or 
were states’ residents who delayed going to college. 


Median earnings for the college graduates in the MLDE dataset who were found working in-state were 
about 15 percent higher than earnings among the students found working elsewhere (in one of the other 
three states in the pilot). 


Method: Secondary analysis of education and workforce data shared across four states. 


Additional Research Opportunities: More analysis of these datasets is needed to show how people are 
moving between education and employment, such as how timing of credential attainment impacts 
employment and earnings, and how subsequent re-enrollment to pursue higher credentials impacts 
employment and earnings. 


Willett, T. & Stern, B. (2007). The difference between knowing the path and walking the path: 
Predicting student persistence in science from community college to the university. Presentation at CCC 
Student Success Conference. Available from TEBC. 


Key Relevant Findings: Although focused on the transition from CC to university, the report found 


¢ Ethnicity generally not associated with path switching 

* Gender strongly associated with path switching 

* Last community college math course a good predictor of degree path 

* Including other course work strengthens predictions but complicates model 
Method: descriptive statistical analysis of various secondary datasets 


Zinth, J. (2015). Aligning K-12 and postsecondary career pathways with workforce needs. Denver, CO: 
Education Commission of the States. 


Key Relevant Findings: Workforce data is being used to identify disciplines for pathways. There is 
legislation in some states promoting this. 


Method: This review is a compilation of policymaking activity related to secondary- and postsecondary- 
level career/technical education (CTE) in 13 states. 
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Appendix 3: Overarching and Individual Proposal Templates 


PROPOSAL DEVELOPMENT TEMPLATE 
1) OVERARCHING RESEARCH QUESTION: 


2) SPECIFIC RESEARCH SUB-QUESTIONS: 
1 


2. 
3. 
4. 


3) METHODOLOGY: What types of data 
collection and analytical methods could be used 
to address these questions? Which sub- 
questions could best be addressed by which 
methods? 


As the graphic below illustrates, define a set of 
interlocking studies that could address the 
research question from different angles. 


Overarching 

Study 1: Research 
Outcomes-based Question 
quantitative study 


Study 4: Design- 
based Intervention 
Tool Development 
Specific Research 


Specific Research Question 5 


Questions 1 and 2 


Study 3: Design- 
ase 
intervention: PD 


Study 2: Case 
Studies 


Specific Research 
Questions 3 and 4 


Specific Research 
Questions 5&6 


4) SITE/CASE SELECTION CRITERIA: For 
each of the studies proposed, discuss site/case 
selection criteria. Use the expertise of the 
practicing scholars to build proposals grounded 
in rich learning contexts. 


INDIVIDUAL PROPOSAL TEMPLATE 
PROPOSAL TITLE: 
RESEARCH QUESTION(S): 


IMPLICATIONS FOR POLICY & PRACTICE: 


METHODOLOGY: 
* What types of data collection and 
analytical methods do you plan to use? 
¢ How is this research connected to other 
research on the same overarching 
question? 


CASE/SITE SELECTION CRITERIA: If 
applicable, include specific site/context 
proposals 


KEY ACTIVITIES: 


KEY PARTNERS: 


PLAN FOR DEVELOPMENT OF LOGIC 
MODEL, REFINED PROPOSAL, AND 
ONGOING COLLABORATION WITH 
WORKING GROUP. 
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Appendix 4: Group Proposals 
1. ENGAGING STAKEHOLDERS THROUGH DATA 
Proposal Development Template 
1) OVERARCHING RESEARCH QUESTION: 


Can inquiry by students, parents, educators, community members, and other stakeholders 
promote more equitable access and outcomes in pathways in high schools? 


How can 
stakeholders 
access data to 
answer their 

OWN questions? 


Engage students in PAR Educator-led inquiry at 


at their schools to 
increase agency. 


individual school sites 


Case studies of 3 or 4 identified Community/employer 


inquiry 


communities in CA - document 
data and stories already 
collected and how to increase 
data and how to share 
effectively 


2) SPECIFIC RESEARCH SUB-QUESTIONS: 
¢ How can we “the researcher” look at data differently + flip it. School system as place of inquiry. 
* What are existing assets and data platforms that can be used for inquiry? 
¢ Students: What do students think should be measured? “Course” image. 
¢ Teachers & Project Staff 
¢ Parents: Using data at local level — doing research and promoting findings 


* Humanizing the data and sharing the story: helping parents and students connect the dots with 
existing data 


Individual Proposal — Engaging Parents in Pathway Inquiry and Advocacy through Data 


Research Question: Can inquiry by parents promote more equitable access and outcomes in pathways in 
high schools? 


Implications for Policy & Practice: 
e Increase parent engagement in their children’s schools 
¢ Empower parents to advocate for their children 
¢ Develop parent to parent connections around common goals 
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¢ Increase equitable pathway programming and access within a given district 
¢ Increase the number and quality of pathway options for all students within a given district 


Methodology: 
¢ Data collection and analytical methods: 

o Enlist parents in formulating research questions for their children’s’ inquiry into 
pathways and also serving as respondents’ in their children’s’ research (e.g., bringing the 
parent perspective on pathways into their children’s’ research as interviewees). 

o Work with parents to help them make sense of available quantitative and qualitative data 
on pathways (drawing on research and analysis conducted by their own children). 

o Work with parents to help them develop and share stories of their children’s’ experiences 
in pathways, and provide them with data for advocating for their children with their local 
school boards (e.g., through LCAP stakeholder engagement process) and for bringing 
other parents on board with the pathway approach. 

¢ Connection to other research on the same overarching question: 
o This will be connected to the student research on the same topic. 


Case/Site Selection Criteria: 
¢ We will select X number of California districts with pathways as the focus of this work. We will 
work with our partner organizations, such as the state PTA, to identify local PTA groups that 
would be interested in being a part of this effort. 
¢ Feedback: Many parents are engaged. Could take place in a PTA setting, for example. 


Key Activities: 
¢ Identify the districts with pathways that will be the target of our research 
¢ Identify the schools/districts in which the student research component will take place. 
¢ Identify additional partners beyond the state PTA. 


Key Partners: State PTA, PIQE (Parents for Quality Education in San Diego), OCO (Oakland 
Community Organizations)/PICO, Parent Revolution, local school districts willing to collaborate and 
share data 


Plan for Development of Logic Model, Refined Proposal, and Ongoing Collaboration: 
¢ Student effort has to take place first and the logic model should encompass the other related 
stakeholder engagement work. 
¢ The student research proposal has to be developed first to inform the parent engagement 
component. 


In addition, the Engaging Stakeholders through Data subgroup prepared proposal templates for studies 
involving two other stakeholder groups—students and the larger community: 


Individual Proposal — Students Engaging through Data 


Research Question: Can inquiry by students promote more equitable access and outcomes in college and 
career readiness in high schools? 


Implications for Policy & Practice: 
1. Uncover and elevate issues of inequity and work toward more equitable policies and practices at 
the school, district, and state level 
2. Strengthen student agency for the students participating in the program and their peers 


Ee) 


3. Change practices and culture in schools to expect that students have an active voice in decision 
making 
Issues: Data access; old data that is not aggregated 


Methodology: 
1. Literature review around student participatory action research - Review previous examples 
a. Center X 


2. Review existing data sets and/or platforms that can engage students 
3. Curriculum development (Social justice / researcher pathway) 
Theory 
Data 
Data analysis / statistical analysis 
Asking questions / initial findings 
Qualitative data collection — focus groups, interviews 
Presenting findings to multiple audiences — advocacy lens 
i. Peers 
ii. Educators 
ili. Policy maker - School administration, District administration 

4. Implementation (& refinement) 

a. High school 

b. Nonprofit 

c. In partnership with post-secondary (college credit / dual enrollment) 
5. Data collection on impact 

a. Qualitative data — mindset 

i. Student surveys / interviews 
ii. Educator surveys / interviews 
ili. Policy makers surveys / interviews 
b. Document analysis -- Do policies and practice documents change? 
c. Data change over time — Do student opportunities and outcomes change? 


monogs 


Individual Proposal — Engaging the Community through Data 


Research Question: Can inquiry by community members including civic, employer, and postsecondary 
promote more equitable access and outcomes in pathways in high schools? 


Implications for Policy & Practice: 
To expand stakeholder involvement in using data to improve equitable access to college and career 
readiness with long-term goal of impacting policy at the state level. 


Methodology: 

¢ Data collection and analytical methods: 
Use existing data 
National landscape review to identify existing efforts 
Learn more about identified programs/practices 
Identify sites in CA with existing programs/practices to determine what data they are 
analyzing and how they are disseminating and using it to create more equitable access 
and researchers support them to broaden the scope of data used and their outreach 
capacity by documenting their practice and identifying ways to tell their story to a wider 
audience. 
* Connection to other research on the same overarching question: 

o Communication across stakeholder groups (parents, students, teachers) to share data 


oO°0 0 
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Case/Site Selection Criteria: 

* Invite communities in CA that were identified in the landscape review that express interest in 
refining and increasing activities to promote equitable access to college and careers based on 
data. 

¢ Invite communities that expressed in pursuing conversations and activities to use data to promote 
equitable access to college and careers. 


Key Activities: 
¢ Survey the national landscape to identify communities where community members are already 
engaging in effective conversations using data to achieve equitable access. 
¢ Identify communities interested in pursuing conversations and activities to use data to promote 
equitable access to college and careers. 


Next Steps: The Engaging Stakeholders through Data subgroup met on Nov. 16, 2017. They discussed 
whether to develop separate proposals for each stakeholder group (i.e., students, parents) or one proposal 
including all stakeholder groups. They decided that there might be greater support for a comprehensive 
proposal examining the various ways that different stakeholders would identify the information they need, 
learn where to get it, and produce something that would have impact related to their specific concerns. 
Also, logistically, the difficulty of accessing school data might justify having all stakeholders under one 
study umbrella. Members described their current work and how it related to the subgroup topic. One 
noted new work being done in her K-12 school district to build a course piloted through the local UC 
campus that plans to utilize and leverage district data. She thought that her colleagues would like to have 
research partners in the work. Before the end of 2017, the convener agreed to draft a statement about what 
this group is about and what they would like to do. 


2. HIGH QUALITY COLLEGE AND CAREER PATHWAYS 


Overarching Question: What is the impact of high quality college and career pathways, for whom and 
under what conditions? 


Research Questions: 

¢ What is a high quality career pathway? 

* What are the essential components or combination of components that are essential for high 
quality pathways (and interaction effects among the components)? 

¢ What does it take to implement the essential components? 

¢ What are the supplemental components over and above the essential components? 

¢ What is the needed dosage within each component to achieve desired results? 

¢ (sidebar question) What is the return on investment (ROJ) of the program for employers and the 
district? 


Methodology: 

Begin by reviewing documentation about existing high quality programs. It might be difficult to construct 
an experimental study, so consider beginning with small descriptive and comparative case studies, 
perhaps isolating specific pathways components. 


Individual Proposal — High Quality College and Career Pathway Components 


Research Question: What are the essential components or combination of components that are essential 
for high quality pathways (and interaction effects among the components)? 
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Methodology: 


Comparative case study approach -- look at variation and the antecedents -- exploring naturally 
occurring variation 

Additive experimentation -- experimental variation - like qualitative A-B testing -- testing 
different implementation strategies simultaneously. Qualitative researchers would inventory 
sites; sites would then be offered additional strategies and support implementation; then test the 
variation (induced variation); a test of coaching or professional development (based on the 
inventory, components or dosage would be added) 

Notes: This would not be starting at ground zero (Like ECCO project), it could start with small 
interventions, using a pre-post model, try to randomize schools 


Case/Site Selection Criteria: 


Need to look at outcome and component pairs 
Need to look at the outcomes that are very close to the treatment, at the implementation level; 
Take the measure after the intervention; pre-post, comparative, around the additional components 
(e.g., WBL) as the value added intervention, with 21st Century skills as one of the outcomes that 
need to be measured 
Offer to implement ECCO (which includes working closely with supervisors to ensure they are 
supporting behaviors that are being measures, such as collaboration) at two schools; the impact is 
increased internships as the output -- are more internships are being offered; then examine student 
outcomes 
Need to pair with student outcomes -- did ECCO intervention? 
Comparative time series; mixed method; follow-up would use assessments and look at 
sustainability (a Phase Three) 
Site selection: Programs that are in the “middle of the pack” 
o Start with sites with cups partly full; based on self-assessment (e.g. NAF pathway 
assessment) 
o What does it take to turn a school rated 4 into one rated 6 (find components that do not 
have a large fixed cost) 


Key Activities: 


Needs assessment: design the intervention 
Data collection: document review, field work, collection of survey data 


Key Partners: Districts, pathway intermediaries 


Next Steps: This subgroup will meet on Dec. 13, 2017. 


3. MEASUREMENT 


Overarching Question: What are the measures of engagement and short-term outcomes associated with 
pathways students’ long-term success? 


Research Questions: 


What are the jobs of the future and the skills they demand? 

How can we measure the benefits of employers’ participation in career pathways? 
o ROI 

o Employers’ perceptions 
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¢ What are the measures and milestones of students’ engagement in a pathway experience that lead 
to life success? 

¢ What are the constructs we should use to measure career readiness that are generalizable across 
occupation types? Sector-specific? 

* How should we measure whether students are developing 21st Century skills that are in demand 
by the workforce? 


o Mindsets 

o Self-regulation 

o Intra 

o Inter 

o Teamwork, collaboration 


Individual Proposal — Measures and Milestones of Student Engagement: Implementation Study 


Research Question: 
1. What are the key components (behaviors, mindsets, values, abilities) of engagement in a high 
school pathway that predict future life success (college and career readiness)? 
a. What are the best sources of evidence (extant data, tasks, situations) of engagement? 
b. How do we interpret evidence of engagement for the purpose of identifying impact of 
pathway experiences on engagement? 
2. What aspects of the pathway experience are associated with high-levels of student engagement? 


3. What strategies for implementing pathways lead to high-levels of student engagement? 
4. What are the key milestones that indicate high-levels of student engagement in a high school 
pathway? 
a. What, if any, are important early indicators (first or second year) of engagement in 
pathway? 


b. What, if any, are important late indicators (third or fourth year)? 


Implications for Policy & Practice: 
¢ Results would inform efforts to develop constructs and measures of student engagement in 
pathways that could be used to evaluate and improve quality of pathways. 
¢ Better measures of student engagement in pathways could be used to evaluate and improve 
pathway design and implementation 


Methodology: 
¢ Identify cohorts of students across multiple sites currently enrolled in pathways 
¢ Survey and interview about engagement, interest, etc. 
¢ Explore engagement and career-readiness construct development 
¢ Follow-progression forward in time 


Case/Site Selection Key Activities and Partners: The group discussed various available datasets such 
as NAF, California Partnership Academy, and SRI datasets. 


Individual Proposal — Measures and Milestones of Student Engagement: Retrospective Study 
Research Question: What are the measures and milestones of students’ engagement in a pathway 
experience that lead to life success? What effect does student engagement in a high school pathway have 


on postsecondary and career success? 


Sub-questions: 
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1. Did successful adults who completed a high school pathway experience higher levels (or different 
forms) of engagement in high school compared with comparable adults who were not in a 
pathway? 

a. Do successful adults who were in a high school pathway say they experienced a high- 
level of engagement in the pathway? What was the nature of their engagement (or lack of 
engagement) and what aspects of the pathway experience do they feel best explain their 
level of engagement? 

b. In what ways was the high school engagement of a comparable group of adults who were 
not in a high school pathway similar or different from that of pathway completers? 

2. What are the implications of the experience of successful adults for identifying key components 
(behaviors, mindsets, values, abilities) of engagement in a high school pathway that are 
associated with future life success? 


Implications for Policy & Practice: 
* Results would provide evidence of the association of student engagement in high school 
pathways with positive college and career trajectories. 
* Results would inform efforts to develop constructs and measures of student engagement in 
pathways that could be used to evaluate and improve quality of pathways. 


Methodology: 
¢ Data collection and analytical methods: 
o Identify students who have already graduated from a pathways experience and are in 
college and careers 
o  Interview/survey students about reflections on pathway experience and what mattered 
o Use extant data to analyze students’ progression through pathways and examine what 
predicts students’ long-term outcomes 


* Connection to other research on the same overarching question: 
o A retrospective study could inform development of an engagement construct to guide design 
of measures of engagement to be used in other studies 
o Depending on sequencing of studies, a retrospective study could also be used to validate 
measures of engagement 


Next Steps: The measurement subgroup met on a conference call on Nov. 21, 2017. Between the 
symposium and the conference call, members drafted a logic model for the implementation proposal. 
Members agreed to refine the key research questions and fill out the Individual Proposal template for 
them. The most developed ideas will be put on Google Drive so they can be accessed and edited by all. 
The group discussed methodology, site selection, and the types of data needed to develop indicators and 
answer questions on student engagement. 


4. POLICY 
Overarching Question: How are school districts leveraging local, state, and federal policy to develop, 
implement, and sustain high quality college and career pathways that serve students to and through 
postsecondary education? 


Individual Proposal — Interpreting College and Career Pathways Policy at the Local Level 


Research Question: 
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How can a school district’s vision for college and career pathways drive local interpretation of 
local, state, and federal policy? 

How can policy be changed to promote the integration of rigorous academics and career and 
technical education? 


Implications for Policy & Practice: 


Community civic engagement and leadership around college and career pathways were 
hypothesized to define a local vision that drives policy interpretation and impacts college and 
career pathway implementation. 

This study would trace the impact of vision on a few key components of implementation: 
equitable access to postsecondary options, and the extent to which rigorous academics and career 
and technical education are actually integrated. 

This study could examine the range, character, and priorities embedded in visions for college and 
career pathways, developed by engaged community leaders in various contexts, and could 
investigate the links between their vision and their interpretation of local, state and federal policy. 
This would have value for district leaders trying to figure out how to implement college and 
career pathways, as it would development of leadership that can leverage local, state and federal 
policies. 


Methodology: A mixed-methods study is called for, including: 


Document analysis (e.g., local control plans and process documentation), 

A survey of district leaders, particularly those involved in the development of district vision 
statements, and 

Case studies of specific districts including interviewing school board and PTA members, district 
staff and leaders, and site and pathway leaders. 


Case/Site Selection Criteria: Case selection would focus on districts with a stated commitment to 
college and career pathways, and to integrating academics and career and technical education. Districts 
serving highly vulnerable, underserved populations would also be prioritized. 


Next Steps: Because this proposal meshes well with the capacity building symposium scheduled for 
2018, this subgroup decided that the proposal would be further developed at that time. 


5. EQUITY AND ACCESS 


Overarching Question: What does equitable access to pathways look like and how does it work? 


Research Questions: 


To what degree is there equitable access to high quality college and career pathways? 
o What students have access to what pathways — themes and quality of the pathway? 
How does the process work? 
o What do schools do to inform students to make decisions about pathways and what role 
do state and local policies play? 
o How do students and families make decisions about pathways? 
o How do these patterns vary by gender, race, and subgroup status (socioeconomic status, 
EL, migrant students, etc.) 
Process questions: 
o What is the process by which students make decisions about selecting a pathway? 
o To what extent is the counselor, parent, teacher, student, etc. involved? 
o How does the school structure that process? 
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Is there a selection criteria? 
Are these processes differential as it relates to who has access to information? 
Does the location of a pathway matter in the decision-making process? 
o When do students begin to make decisions? (Exposure and awareness) 
¢ Equity lens: 
o Who is in what pathway? Race, socioeconomic status 
o What can make a difference? 


oOo Oo 


Methodology: 
¢ Begin by describing the status quo: examine state-level data to see which pathways currently exist 
where 


¢ Case study — LBUSD and 2 other districts 
¢ Multi-case study comparing pathway choice process across districts. 
¢ Access study: Is there a difference across districts? 


Case/Site Selection Criteria: Districts must be willing to provide data 


Next Steps: As noted above, equity issues came to be seen as foundational to all subgroup research 
ideas. The Equity and Access subgroup developed ideas that they then found embedded explicitly or 
implicitly in other subgroups’ proposals. At the final gallery walk, everyone saw that attention to equity 
had to inform and shape how each research proposal was framed. The foundational work by this subgroup 
is helping the Planning Committee reframe the next symposium on teaching and learning in pathways: the 
entire symposium will use an equity lens to examine pedagogical issues in college and career pathways. 
Equity is not a stand-alone issue but rather an overarching one that influences all aspects of pathways and 
pathways research. Some of the issues raised by this group will be reserved for a more focused 
examination at a later symposium. 
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ABOUT 


About the Collegeand Career Pathways Research Symposia Series 


CCASN, in conjunction with other nationally renowned research-based organizations, is hosting a 
series of four symposia, each of which will convene experts on topics related to college and career 
pathways. We will map what we know, identify gaps in the research, and pinpoint key strategic foci 
for moving policy and practice forward. The four symposium topics are: 1) the secondary student 
experience, 2) equity issues in teaching and learning, 3) leadership and capacity building, and 4) 


systems alignment. 


The second symposium, which will address Equity Issues in Teaching and Learning in Pathways, is 


currently being planned for April 2018. 


The planning committee for the symposia series includes representatives from the following 


organizations: 


ASAS N 


College & Career Academy Support Network 


UNIVERSITY OF CALIFORNIA, BERKELEY 
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